Eye opening degree detection system, doze detection system, automatic shutter system, eye opening degree detection method, and eye opening degree detection program

ABSTRACT

An object of the present invention is to provide an eye opening degree detection system that can accurately and stably calculate an eye opening degree. The eye opening degree detection system includes imaging devices that generate images including regions of both eyes, a one eye opening degree calculation unit that calculates each one eye opening degree of the left and right eyes of a first image, an eye opening degree selection unit that selects a correctly-calculated one eye opening degree, an eye opening degree calculation unit that calculates an eye opening degree on the basis of the one eye opening degree and an eye opening degree determination unit that compares the eye opening degree calculated on the basis of the first image with an eye opening degree calculated on the basis of a second image prior to the first image to determine the propriety of the eye opening degree.

CROSS-REFERENCE TO RELATED APPLICATIONS

The application is a Continuation of U.S. patent application Ser. No.15/428,680, filed on Feb. 9, 2017, which claims the benefit of JapanesePatent Application No. 2016-025839 filed on Feb. 15, 2016 including thespecification, drawings and abstract is incorporated herein by referencein its entirety.

BACKGROUND

The present invention relates to an eye opening degree detection system,a doze detection system, an automatic shutter system, an eye openingdegree detection method, and an eye opening degree detection program.

An eye opening degree detection system for detecting dozing of a driverof a vehicle has been developed. For example, Japanese Unexamined PatentApplication Publication No. 2004-041485 describes an eye opening/closingmonitoring device that sets a threshold value for determining eyeclosing irrespective of individuals or environments.

SUMMARY

In the eye opening degree detection system according to the related art,when detecting an eye opening degree on the basis of an image imaged bya camera, blinks are determined by changing a threshold value fordetermining eye opening and closing in a bad light environment. However,there is a possibility that the eye opening degree detected using theimage imaged in a bad light environment is inaccurate even if thethreshold value for determining eye opening and closing is changed, andthus the determination of dozing becomes unstable.

Accordingly, an eye opening degree detection system, a doze detectionsystem, an automatic shutter system, an eye opening degree detectionmethod, and an eye opening degree detection program that can accuratelyand stably calculate the eye opening degree have been desired.

The other objects and novel features will become apparent from thedescription of the specification and the accompanying drawings.

According to an embodiment, provided is an eye opening degree detectionsystem that includes: a plurality of imaging devices that generatesimages including, at least, regions of both eyes of a target personwhile shifting time; a one eye opening degree calculation unit thatcalculates each one eye opening degree of the left and right eyes of afirst image; an eye opening degree selection unit that creates a pixelvalue histogram of each region of the left and right eyes when the leftand right one eye opening degrees do not substantially match each other,and selects a correctly-calculated one eye opening degree on the basisof the pixel value histogram; an eye opening degree calculation unitthat calculates an eye opening degree on the basis of the left and rightone eye opening degrees when the left and right one eye opening degreessubstantially match each other, and calculates an eye opening degree onthe basis of the one eye opening degree selected by the eye openingdegree selection unit when the left and right one eye opening degrees donot substantially match each other; and an eye opening degree detectiondevice having an eye opening degree determination unit that compares theeye opening degree calculated on the basis of the first image with aneye opening degree calculated on the basis of a second image prior tothe first image to determine the propriety of the eye opening degreecalculated on the basis of the first image.

It should be noted that ones expressed by replacing the system of theabove-described embodiment with a device, a method, or another system(for example, a doze detection system or an automatic shutter system), aprogram that allows a computer to execute a process of the system or apart of the system, and an imaging device and an automobile having thesystem are applicable as modes of the present invention.

According to the above-described embodiment, it is possible to providean eye opening degree detection system, a doze detection system, anautomatic shutter system, an eye opening degree detection method, and aneye opening degree detection program that accurately and stablycalculate an eye opening degree.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram for showing an outline configuration of a dozedetection system 1 according to a first embodiment;

FIG. 2 is a flowchart for showing a processing procedure of a dozedetection method according to the first embodiment;

FIGS. 3A and 3B are diagrams each explaining a calculation method of aone eye opening degree a according to the first embodiment;

FIG. 4 shows an example of a brightness value histogram according to thefirst embodiment;

FIG. 5 is a diagram for explaining a determination method ofcharacteristics of cameras according to the first embodiment;

FIG. 6 shows an example of an eye opening degree transition graphaccording to the first embodiment;

FIGS. 7A, 7B and 7C show other examples of the eye opening degreetransition graph according to the first embodiment;

FIG. 8 is a diagram for showing a state in which the doze detectionsystem 1 according to the first embodiment is attached to the inside ofan automobile 6;

FIG. 9 is a diagram for showing an outline configuration of an automaticshutter system 2 according to a second embodiment;

FIG. 10 is a flowchart for showing a processing procedure of anautomatic shutter method according to the second embodiment; and

FIG. 11 shows an example of an eye opening degree transition graphaccording to the second embodiment.

DETAILED DESCRIPTION

In order to clarify the explanation, the following descriptions anddrawings are appropriately omitted and simplified.

Further, the same reference numerals are given to the sameconstitutional elements in the respective drawings, and duplicatedexplanations are omitted as needed.

First Embodiment

A doze detection system according to a first embodiment is mounted in avehicle such as an automobile, a target driver is imaged by a pluralityof cameras while shifting time, and dozing of the driver is detected onthe basis of the eye opening degrees of the eyes of the drivercalculated using the images.

First, a configuration of a doze detection system 1 according to thefirst embodiment will be described.

FIG. 1 is a diagram for showing an outline configuration of the dozedetection system 1 according to the first embodiment.

The doze detection system 1 includes two cameras 11 and 12, an eyeopening degree detection unit 20, a graph creation unit 30, a dozedetection unit 40, an alarm device 50, and the like.

The cameras 11 and 12 are arranged diagonally forward on the left andright sides of a driver to alternately image the driver, and output theimages to the eye opening degree detection unit 20.

The eye opening degree detection unit 20 detects an eye opening degreeat each timing when each of the images was imaged on the basis of theimages. Therefore, the eye opening degree detection unit 20 includes aone eye opening degree calculation unit 21, an eye opening degreeselection unit 22, an eye opening degree calculation unit 23, an eyeopening degree determination unit 24, and the like.

The one eye opening degree calculation unit 21 extracts each region ofthe left and right eyes from each image, and calculates a feature amountto calculate each eye opening degree (hereinafter, referred to as a “oneeye opening degree”) of the left and right eyes. Further, the one eyeopening degree calculation unit 21 determines whether or not thecalculated left and right one eye opening degrees substantially matcheach other, and outputs the result to the eye opening degree selectionunit 22 and the eye opening degree calculation unit 23.

When the left and right one eye opening degrees calculated by the oneeye opening degree calculation unit 21 do not substantially match eachother, the eye opening degree selection unit 22 creates a pixel valuehistogram of each region of the left and right eyes, and selects acorrectly-calculated one eye opening degree on the basis of the pixelvalue histogram to output the result to the eye opening degreecalculation unit 23.

When the left and right one eye opening degrees calculated by the oneeye opening degree calculation unit 21 substantially match each other,the eye opening degree calculation unit 23 calculates the eye openingdegree on the basis of the left and right one eye opening degrees.Further, when the left and right one eye opening degrees do notsubstantially match each other, the eye opening degree is calculated onthe basis of the one eye opening degree selected by the eye openingdegree selection unit 22. In addition, the calculated eye opening degreeis output to the eye opening degree determination unit 24.

The eye opening degree determination unit 24 compares the eye openingdegree of the image imaged at certain timing with the eye opening degreeof the image imaged at preceding timing. When it is determined that theeye opening degree at the certain timing is correct, the eye openingdegree at the certain timing is output to the graph creation unit 30.

The graph creation unit 30 creates an eye opening degree transitiongraph showing a relation between the imaging timing and the eye openingdegree of each image, and outputs the graph to the doze detection unit40.

The doze detection unit 40 determines a period of time (hereinafter,referred to as an “eye opening period of time”) during which the eyes ofthe driver are opened and a period of time (hereinafter, referred to asan “eye closing period of time”) during which the eyes of the driver areclosed on the basis of the eye opening degree transition graph, anddetects dosing of the driver on the basis of a ratio of the eye openingperiod of time to the eye closing period of time in a predeterminedperiod of time. Further, when detecting dosing of the driver, the dozedetection unit 40 outputs the result to the alarm device 50 such as aspeaker, and the alarm device 50 generates an alarm for the driver.

With such a configuration, the doze detection system 1 can accuratelyand stably detect the eye opening degree, and can properly detect dosingof the driver on the basis of the detected eye opening degree.

It should be noted that each constitutional element realized by the dozedetection system 1 can be realized by executing a program under thecontrol of, for example, a computing device (not shown) included in thedoze detection system 1 (excluding the cameras 11 and 12 and the alarmdevice 50) that is a computer.

More specifically, the doze detection system 1 realizes eachconstitutional element in such a manner that a program stored in astoring unit (not shown) is loaded into a main storing device (notshown) and is executed under the control of the computing device.Further, it is not necessary to realize each constitutional elementusing a software program, but each constitutional element may berealized using any combination of hardware, firmware, and software.

The above-described program is stored using non-transitory computerreadable media of various types, and can be supplied to the dozedetection system 1. The non-transitory computer readable media includetangible storage media of various types.

Examples of the non-transitory computer readable media include amagnetic recording medium (for example, a flexible disk, a magnetictape, or a hard-disk drive), a magnetooptic recording medium (forexample, a magnetooptic disk), a CD-ROM (Read Only Memory), a CD-R, aCD-R/W, and a semiconductor memory (for example, a mask ROM, a PROM(Programmable ROM), an EPROM (Erasable PROM), a flash ROM, or a RAM(random access memory)).

Further, the program may be supplied to the doze detection system 1 bytransitory computer readable media of various types. Examples of thetransitory computer readable media include an electric signal, anoptical signal, and an electromagnetic wave. The transitory computerreadable media can supply the program to the doze detection system 1through a wired communication channel such as an electric wire or anoptical fiber, or a wireless communication channel.

Next, an operation of the doze detection system 1 according to the firstembodiment, namely, a doze detection method will be concretelydescribed.

FIG. 2 is a flowchart for showing a processing procedure of the dozedetection method according to the first embodiment.

When an operation of the doze detection system 1 is started, the cameras11 and 12 alternately image the driver (Step S10). In this case, eachshutter speed of the cameras 11 and 12 is set to, for example, about 30ms (30 fps).

Next, the one eye opening degree calculation unit 21 extracts eachregion of the left and right eyes from one of the images (Step S20). Amethod of extracting the regions of the left and right eyes may be anyone of well-known methods. However, for example, a method described inJapanese Unexamined Patent Application Publication No. 2008-191785 isused.

Next, a one eye opening degree a is calculated in each region of theleft and right eyes extracted by the one eye opening degree calculationunit 21 (Step S30).

FIGS. 3A and 3B are diagrams each explaining a calculation method of theone eye opening degree a according to the first embodiment. FIG. 3Ashows a shape of an eye that is normally opened, and FIG. 3B shows ashape of an eye in the middle of blinking. An eye opening width K isdefined by the number of pixels (or a dimension) between the uppermostpart and the lowermost part of upper and lower eyelids.

The one eye opening degree calculation unit 21 preliminarily calculatesthe eye opening width Kmax of the eye that is normally opened, andcalculates the eye opening width K of the eye extracted in Step 20 tocalculate the one eye opening degree a (=K/Kmax). In this case, the oneeye opening degrees of the right and left eyes are represented by αR andαL, respectively.

In addition, the one eye opening degree calculation unit 21 determineswhether or not the left and right one eye opening degrees αL and αRsubstantially match each other (Step S40). In this case, thedetermination can be made on the basis of whether or not, for example,the difference between the left and right one eye opening degrees αL andαR falls within a predetermined ratio of the one eye opening degree athat is larger than another.

Next, when it is determined that the left and right one eye openingdegrees αL and αR do not substantially match each other (No in StepS40), the one eye opening degree calculation unit 21 determines that theeye opening degree is unstable due to an abnormal light environment inany one of the regions of the left and right eyes or an abnormal lightenvironment of any one of the cameras 11 and 12 deriving from sunlightor headlights at night. Thus, the eye opening degree selection unit 22creates a pixel value histogram of the regions of the left and righteyes, for example, a brightness value histogram to determine the stateof the light environment (Step S50).

FIG. 4 shows an example of the brightness value histogram according tothe first embodiment.

The histogram R of the region of the right eye mostly falls within anormal value region, and the histogram L of the region of the left eyemostly falls within an abnormal (bright) region. Namely, it can beassumed that intense light such as sunlight enters from the leftdirection.

Accordingly, the eye opening degree selection unit 22 determines thatthe one eye opening degree αL is not correctly calculated due to theabnormal light environment of the left eye, and the one eye openingdegree αR is correctly calculated due to the normal light environment ofthe right eye (Step S60).

In addition, when the eye opening degree selection unit 22 determinesthat the light environment of either the left eye or the right eye isnormal, namely, the light environment of the right eye is normal in thiscase (Yes in Step S60), the eye opening degree calculation unit 23calculates an eye opening degree βA on the basis of the followingequation (1) (eye opening degree calculation equation) (Step S70).

βA=AR×αR+AL×αL   Equation (1)

Specifically, when the light environment of the right eye is normal andthe light environment of the left eye is abnormal, the eye openingdegree calculation unit 23 sets an eye opening degree state coefficientAR of the right eye in the eye opening degree calculation equation to 1,and sets an eye opening degree state coefficient AL of the left eye thatis an unstable element to 0 as shown in the following equation (2).

βA=1×αR+0×αL=αR   Equation (2)

Namely, the one eye opening degree αR is set to the eye opening degreeβA to stabilize the eye opening degree βA.

On the other hand, when it is determined that the left and right one eyeopening degrees αL and αR substantially match each other (Yes in StepS40), the one eye opening degree calculation unit 21 determines that thelight environments of both eyes are normal and the eye opening degreesare stable. Thus, the eye opening degree calculation unit 23 sets eachof the left and right eye opening degree state coefficients AL and AR inthe eye opening degree calculation equation to 0.5 to calculate the eyeopening degree βA as shown in the following equation (3).

βA=0.5×αR+0.5×αL   Equation (3)

In addition, when the light environments of both eyes are not normal (Noin Step S60), or when the eye opening degree βA is calculated (Step S70and Step S80), it is determined whether or not the eye opening degreecalculation unit 23 can calculate the eye opening degree βA and theabsolute value |βA−βB| of the difference between the eye opening degreeβA calculated this time and an eye opening degree βB calculatedimmediately before is 0.3 or smaller (Step S90).

In the doze detection method according to the first embodiment, thecameras 11 and 12 alternately image a plurality of images, andcalculates the eye opening degree β at shorter time intervals asdescribed above. On the assumption that the eye opening degreecalculated this time is represented by βA and the eye opening degreecalculated from the last image from which the present eye opening degreeβA was calculated is represented by βB, the abnormality ofcharacteristics of the cameras can be detected by comparing the eyeopening degrees βA and βB with each other because of the reasondescribed below.

FIG. 5 is a diagram for explaining a determination method ofcharacteristics of the cameras according to the first embodiment. Thehorizontal axis of the graph in the drawing represents time t, and thevertical axis represents the eye opening degree 13.

The blink speed of a human being is generally about 100 to 150 ms. Ifthe blink speed is assumed as 100 ms, a period of time required from theeyelid opened state to the eyelid closed state is about 50 ms, and theeye opening degree β at the time is changed by about 1.0. In addition,the eye opening degree β changed in about 15 ms that is a shutter timedifference between the cameras 11 and 12 each having a shutter speed ofabout 30 ms (30 fps) is about 0.3 (=1.0×15/50) or smaller even in thecase of a person whose blink speed is fast. Namely, if the absolutevalue |βA−βB| of the difference between the present eye opening degreeβA and the last eye opening degree βB is 0.3 or smaller, thecharacteristics of the cameras and the light environment of the presentimage are normal. If the absolute value |βA−βB| of the difference islarger than 0.3, the characteristics of the cameras and the lightenvironment of the present image can be determined as abnormal.

When it is determined that the eye opening degree βA is not calculatedor the absolute value |βA−βB| of the difference is not 0.3 or smaller(No in Step S90), the eye opening degree calculation unit 23 determinesthat the eye opening degree βA is abnormal, and sets the present eyeopening degree βA to 0 (Step S100). Then, the flow returns to Step S10.Further, when it is determined that the eye opening degree βA iscalculated and the absolute value |βA−βB| of the difference is 0.3 orsmaller (Yes in Step S90), the eye opening degree calculation unit 23determines that the eye opening degree βA is normal, and sets thepresent eye opening degree βA to the value thereof (Step S110). Then,the eye opening degree βA is stored.

Next, the graph creation unit 30 determines whether or not a certainperiod of time required to create the graph has elapsed (Step S120).When the graph creation unit 30 determines that the certain period oftime has not elapsed (No in Step S120), the flow returns to Step S10.When the graph creation unit 30 determines that the certain period oftime has elapsed (Yes in Step S120), the eye opening degree transitiongraph is created (Step S130).

FIG. 6 shows an example of the eye opening degree transition graphaccording to the first embodiment. The horizontal axis of the graphrepresents time t, and the vertical axis represents the eye openingdegree β.

Next, the doze detection unit 40 separates the eye opening degreetransition graph into eye opening periods of time Na and eye closingperiods of time Nb using a predetermined eye opening/closing thresholdvalue βth (Step S140). In FIG. 5, the eye opening periods of time Na arerepresented by dotted lines, and the eye closing periods of time Nb arerepresented by solid lines.

Next, the doze detection unit 40 calculates an eye opening ratio X(=Na/Nb) that is a ratio of the eye opening period of time Na to the eyeclosing period of time Nb in a predetermined period of time (Step S150).

Next, the doze detection unit 40 also determines whether or not the eyesare excessively opened (X≈∞) and the eyes are excessively closed (X≈0)on the basis of whether or not the eye opening ratio X falls within apredetermined range (Xth1<X<Xth2) (Step S160). When the doze detectionunit 40 determines that the eye opening ratio X falls within thepredetermined range (Yes in Step S160), it is determined that the driveris awaking (Step S170), and the flow returns to Step S10.

Further, when the doze detection unit 40 determines that the eye openingratio X does not fall within the predetermined range (No in Step S160),it is determined that the driver is dozing (Step S180), and an alarm isoutput from the alarm device 50 (Step S190). Then, the flow returns toStep S10.

As described above, in the doze detection system 1 or the doze detectionmethod according to the first embodiment, a target person is imagedusing two cameras 11 and 12, and the eye opening degree β can bestabilized against the light environment by using only the eye openingdegree β in a normal light environment. Further, the cameras 11 and 12can detect the eye opening degrees β in a time series manner by imaginga target person while shifting the imaging time. A change between theeye opening degree βB calculated last and the eye opening degree 13Acalculated this time is comprehensively determined together withenvironmental changes, so that the eye opening degree β can beaccurately and stably detected.

It should be noted that dozing may be detected using three or morecameras in the doze detection system or the doze detection methodaccording to the first embodiment.

For example, a doze detection system that images a target person whileshifting time with three cameras each having a shutter speed of about 30ms (30 fps) can be configured only by adding one camera to the dozedetection system 1 shown in FIG. 1. In addition, it is only necessary tochange the doze detection method to one that uses three cameras in theprocedures shown in FIG. 2.

Namely, if three cameras are used, the shutter time difference amongthose is about 10 ms, and the eye opening degree β changed during theperiod of time is about 0.2 (=1.0×10/50) or smaller even in the case ofa person whose blink speed is fast. Accordingly, if the absolute value|βA−βB| of the difference between the present eye opening degree βA andthe last eye opening degree βB is 0.2 or smaller, the characteristics ofthe cameras and the light environment of the present image may bedetermined as normal. If the absolute value |βA−βB| of the difference islarger than 0.2, the characteristics of the cameras and the lightenvironment of the present image may be determined as abnormal.

Further, in the case where the last eye opening degree βB is 0, if theabsolute value |βA−βC| of the difference between the present eye openingdegree βA and an eye opening degree βC before last is 0.4 or smaller,the characteristics of the cameras and the light environment of thepresent image may be determined as normal. If the absolute value |βA−βC|of the difference is larger than 0.4, the characteristics of the camerasand the light environment of the present image may be determined asabnormal.

Even in the case where one of three or more cameras has a problem, theeye opening degree β can be stably calculated by the remaining cameras.Further, the eye opening degree β of one camera having a problem can beinterpolated using the eye opening degrees β of the remaining cameras.

FIGS. 7A, 7B and 7C show other examples of the eye opening degreetransition graph according to the first embodiment. FIGS. 7A, 7B and 7Cshow transition graphs in which the eye opening degrees β werecalculated using three cameras.

FIG. 7A shows an example of the eye opening degree transition graph inwhich all the eye opening degrees β calculated using three cameras werenormal.

On the contrary, FIG. 7B shows an example of the eye opening degreetransition graph in which some of the calculated eye opening degrees βwere abnormal. The abnormal cases of the eye opening degrees β include acase in which the one eye opening degrees αR and αL do not substantiallymatch each other (No in Step S40), a case in which the lightenvironments of both eyes are not normal (No in Step S60), and a case inwhich the absolute value |βA−βB| of the difference is not 0.2 or smaller(No in Step S90). In the processing procedure shown in FIG. 2, when theeye opening degree βA is abnormal as described above, the eye openingdegree βA is set to 0. Thus, the graph shown in FIG. 7B is not actuallycreated.

FIG. 7C shows an example of the eye opening degree transition graph inwhich the eye opening degree βA is set to 0 in the case as shown in FIG.7B. Even in such a case, the eye opening degree transition graph can beproperly created by the doze detection system according to the firstembodiment.

FIG. 8 is a diagram for showing a state in which the doze detectionsystem 1 according to the first embodiment is attached to the inside ofan automobile 6. The cameras 11 and 12 are attached to a rearview mirror7, and the alarm device (speaker) 50 is attached to the ceiling of theautomobile 6. The distance between the cameras 11 and 12 and the driveris, for example, 60 cm.

The cameras 11 and 12 may be wide-angle cameras (cameras having wideviewing angles). In this case, since the horizontal widths of the leftand right eyes are changed, the one eye opening degree calculation unit21 can correct horizontal width extension (distortion) when each regionof the left and right eyes is extracted from the image (Step S20).

Further, in the doze detection system 1 or the doze detection methodaccording to the first embodiment, dosing may be detected using theinclination when blinking in the eye opening degree transition graph,namely, using the eye opening/closing speed, instead of detecting dosingusing the eye opening ratio X in a predetermined period of time. Thedosing of the driver can be easily detected by using the fact that thevalue of the eye opening/closing speed when the driver is sleepy issmaller than that when the driver is awaking.

As described above, the eye opening degree detection system according tothe first embodiment includes: the imaging devices 11 and 12 thatgenerate images including, at least, regions of both eyes of a targetperson while shifting time; the one eye opening degree calculation unit21 that calculates each of the one eye opening degrees αL and αR of theleft and right eyes of a first image; the eye opening degree selectionunit 22 that creates the pixel value histogram of each region of theleft and right eyes when the left and right one eye opening degrees αLand αR do not substantially match each other, and selects thecorrectly-calculated one eye opening degree a on the basis of the pixelvalue histogram; the eye opening degree calculation unit 23 thatcalculates the eye opening degree β on the basis of the left and rightone eye opening degrees αL and αR when the left and right one eyeopening degrees αL and αR substantially match each other, and calculatesthe eye opening degree β on the basis of the one eye opening degree aselected by the eye opening degree selection unit when the left andright one eye opening degrees αL and αR do not substantially match eachother; and the eye opening degree detection device 20 having the eyeopening degree determination unit 24 that compares the eye openingdegree βA calculated on the basis of the first image with the eyeopening degree βB calculated on the basis of a second image prior to thefirst image to determine the propriety of the eye opening degree βAcalculated on the basis of the first image.

Further, in the eye opening degree detection system according to thefirst embodiment, the eye opening degree determination unit 24preferably determines that the eye opening degree βA calculated on thebasis of the first image is valid when a difference between the eyeopening degree βA calculated on the basis of the first image and the eyeopening degree βB calculated on the basis of the second image is equalto or smaller than a threshold value (for example, 0.3) calculated onthe basis of an imaging timing difference between the first image andthe second image.

Further, the doze detection system 1 according to the first embodimentincludes: the eye opening degree detection system; the graph creationunit 30 that creates the eye opening degree transition graph on thebasis of the eye opening degree βA determined by the eye opening degreedetermination unit 24; and the doze detection unit 40 that determinesthe eye opening period of time Na and the eye closing period of time Nbon the basis of the eye opening degree transition graph, and detectsdozing of the target person on the basis of the ratio X of the eyeopening period of time Na to the eye closing period of time Nb in apredetermined period of time.

Further, when the ratio X is out of a range between the predeterminedlower limit threshold value Xth1 excluding 0 and the predetermined upperlimit threshold value Xth2 excluding infinity, the doze detection system1 according to the first embodiment preferably detects dozing of thetarget person.

Further, the eye opening degree detection method according to the firstembodiment includes: Step S10 of generating images including, at least,regions of both eyes of a target person while shifting time by theimaging devices 11 and 12; Steps S20 to S30 of calculating each of theone eye opening degrees αL and αR of the left and right eyes of a firstimage; Steps S40 to S70 of creating the pixel value histogram of eachregion of the left and right eyes when the left and right one eyeopening degrees αL and αR do not substantially match each other, andselecting the correctly-calculated one eye opening degree a on the basisof the pixel value histogram; Steps S70 and S80 of calculating the eyeopening degree β on the basis of the left and right one eye openingdegrees αL and αR when the left and right one eye opening degrees αL andαR substantially match each other, and calculating the eye openingdegree β on the basis of the selected one eye opening degree a when theleft and right one eye opening degrees αL and αR do not substantiallymatch each other; and Steps S90 to S110 of determining the propriety ofthe eye opening degree βA calculated on the basis of the first image bycomparing the eye opening degree βA calculated on the basis of the firstimage with the eye opening degree βB calculated on the basis of a secondimage prior to the first image.

Further, the eye opening degree detection program according to the firstembodiment allows a computer to execute: Procedure S10 of storing imagesthat include, at least, regions of both eyes of a target person and areobtained by being imaged while shifting time by the imaging devices 11and 12; Procedures S20 to S30 of calculating each of the one eye openingdegrees αL and αR of the left and right eyes of a first image;Procedures S40 to S70 of creating the pixel value histogram of eachregion of the left and right eyes when the left and right one eyeopening degrees αL and αR do not substantially match each other, andselecting the correctly-calculated one eye opening degree a on the basisof the pixel value histogram; Procedures S70 and S80 of calculating theeye opening degree β on the basis of the left and right one eye openingdegrees αL and αR when the left and right one eye opening degrees αL andαR substantially match each other, and calculating the eye openingdegree β on the basis of the selected one eye opening degree a when theleft and right one eye opening degrees αL and αR do not substantiallymatch each other; and Procedures S90 to S110 of determining thepropriety of the eye opening degree βA calculated on the basis of thefirst image by comparing the eye opening degree βA calculated on thebasis of the first image with the eye opening degree βB calculated onthe basis of a second image prior to the first image.

Second Embodiment

The doze detection system 1 using the eye opening degree detection unit20 has been described in the first embodiment. However, an automaticshutter system using the eye opening degree detection unit 20 will bedescribed in a second embodiment.

First, a configuration of the automatic shutter system according to thesecond embodiment will be described.

FIG. 9 is a diagram for showing an outline configuration of an automaticshutter system 2 according to the second embodiment. The automaticshutter system 2 includes a camera 16, an eye opening degree detectionunit 20, a graph creation unit 30, a timing prediction unit 70, and thelike. The automatic shutter system 2 configures, for example, a compoundeye digital camera system in which two cameras are mounted.

The eye opening degree detection unit 20 and the graph creation unit 30may be the same as those according to the first embodiment. Theexplanations of the configurations and operations thereof will beomitted.

The camera 16 includes two imaging units 17 and 18. The imaging units 17and 18 alternately image a target person, and outputs the images to theeye opening degree detection unit 20. Blinking can be stably imaged insome cases by the system in which the two imaging units 17 and 18 areprovided in one camera 16 as compared to a system in which one imagingunit is provided in each of two cameras.

The timing prediction unit 70 calculates an average eye opening periodof time among a plurality of blinks in the past on the basis of the eyeopening degree transition graph created and output by the graph creationunit 30, predicts the start time of the next blink, and determines theshutter time excluding the eye closing period of time. Then, the camera16 automatically releases the shutter on the basis of the shutter timeto image a target person.

Next, an operation of the automatic shutter system 2 according to thesecond embodiment, namely, an automatic shutter method will beconcretely described.

FIG. 10 is a flowchart for showing a processing procedure of theautomatic shutter method according to the second embodiment.

When an operation of the automatic shutter system 2 is started, theimaging units 17 and 18 alternately image a target person, and the eyeopening degree detection unit 20 detects the eye opening degree β. Inaddition, the graph creation unit 30 creates the eye opening degreetransition graph (Step S210). Step S210 corresponds to Step S10 to StepS130 of the processing procedure in the doze detection method accordingto the first embodiment, and the detailed explanation thereof will beomitted (see FIG. 2).

FIG. 11 shows an example of the eye opening degree transition graphaccording to the second embodiment.

Next, the timing prediction unit 70 calculates an average value(hereinafter, referred to as an “average eye opening period of time) ofthe eye opening periods of time Na among blinks in the past (Step S220).In the example of the eye opening degree transition graph shown in FIG.11, the timing prediction unit 70 calculates an average eye openingperiod of time TNa of five eye opening periods of time Na1 to Na5.

Next, the timing prediction unit 70 updates present time T (Step S230).

Next, the timing prediction unit 70 determines whether or not shuttertime TS has been determined (Step S240).

When it is determined that the shutter time TS has not been determined(No in Step S240), the timing prediction unit 70 obtains the present eyeopening degree βA (Step S250).

Next, the timing prediction unit 70 determines whether or not the lastand present eye opening degrees βB and βA are normal values and arelarger than the eye opening threshold value Xth (Step S260).

When the timing prediction unit 70 determines that the last and presenteye opening degrees βB and βA are not normal values or are not largerthan the eye opening threshold value Xth (No in Step S260), the presentimage is determined as a blink state (Step S270), and the flow returnsto Step S230.

On the other hand, when the timing prediction unit 70 determines thatthe last and present eye opening degrees βB and βA are normal values andare larger than the eye opening threshold value Xth (Yes in Step S260),the imaging timing of the present image is set to blink end time T (StepS280).

In addition, the timing prediction unit 70 adds the blink end time TA tothe average eye opening period of time TNa to predict the next blinkstart time TB (TB=TA+TNa) (Step S290), and the shutter time TS(TB=TA+(TNa/2)) excluding the eye closing period of time is determined(Step S300). Then, the flow returns to Step S230.

Further, when the timing prediction unit 70 determines that the shuttertime TS has been determined (Yes in Step S240), it is determined whetheror not the present time T is the shutter time TS (Step S310).

When the timing prediction unit 70 determines that the present time T isnot the shutter time TS (No in Step S310), the flow returns to StepS230.

On the other hand, when the timing prediction unit 70 determines thatthe present time T is the shutter time TS (Yes in Step S310), the camera16 is instructed to release the shutter, and automatically images thetarget person (Step S320). Then, the process is completed.

As described above, the shutter can be automatically released whilemonitoring the blink state even in a bad light environment in theautomatic shutter system 2 and the automatic shutter method according tothe second embodiment.

It should be noted that two imaging units 17 and 18 of one camera 16alternately image a target person, and the images are output to the eyeopening degree detection unit 20 in the automatic shutter system 2 andthe automatic shutter method according to the second embodiment.However, as similar to the first embodiment, two cameras 11 and 12 maybe provided to alternately image a target person, and the images may beoutput to the eye opening degree detection unit 20.

As described above, the automatic shutter system according to the secondembodiment includes the eye opening degree detection system, the graphcreation unit 30 that creates the eye opening degree transition graph onthe basis of the eye opening degrees βA calculated by the eye openingdegree determination unit 24, and the timing prediction unit 70 thatcalculates the average eye opening period of time TNa on the basis ofthe eye opening degree transition graph and determines the shutter timeTS on the basis of the average eye opening period of time TNa and theeye opening degree βA to allow the imaging devices 11 and 12 to image.

The invention achieved by the inventors has been described above indetail on the basis of the respective embodiments. However, it isobvious that the present invention is not limited to the above-describedembodiments, but can be variously changed without departing from thescope of the invention.

What is claimed is:
 1. A data processing device that detects an eyeopening degree based on first and second images that include, at least,regions of both eyes of a target person and that are obtained by beingimaged while shifting time, the data processing device comprising: aprocessor; and a memory accessible by the processor, wherein the memoryaccessible by the processor stores a set of instructions that causes thedata processing device to: calculate each one eye opening degree of theleft and right eyes of the first image; create a pixel value histogramof each region of the left and right eyes when a difference between theleft and right one eye opening degrees does not fall within apredetermined ratio, and selecting one of the left and right one eyeopening degrees based on the pixel value histogram; calculate an eyeopening degree based on the left and right one eye opening degrees whenthe difference between the left and right one eye opening degrees fallswithin the predetermined ratio, and calculating the eye opening degreebased on the selected one eye opening degree when the difference betweenthe left and right one eye opening degrees does not fall within thepredetermined ratio; and determine a propriety of the eye opening degreecalculated based on the first image by comparing the eye opening degreecalculated based on the first image with an eye opening degreecalculated based on the second image prior to the first image.
 2. Thedata processing device according to claim 1, wherein the set ofinstructions further causes the data processing device to compare thecalculated left and right one eye opening degrees of the first image toidentify a larger one eye opening degree, and wherein the predeterminedratio comprises a predetermined ratio to the identified larger one eyeopening degree.
 3. The data processing device according to claim 1,wherein the set of instructions further causes the data processingdevice to determine that the eye opening degree calculated on the basisof the first image is valid when a difference between the eye openingdegree calculated on the basis of the first image and the eye openingdegree calculated on the basis of the second image is equal to orsmaller than a threshold value calculated on the basis of an imagingtiming difference between the first image and the second image.
 4. Thedata processing device according to claim 1, wherein the set ofinstructions further causes the data processing device to: create an eyeopening degree transition graph on the basis of the determined proprietyof the eye opening degree; determine an eye opening period of time andan eye closing period of time on the basis of the eye opening degreetransition graph; and detect dozing of the target person on the basis ofa ratio of the eye opening period of time to the eye closing period oftime in a predetermined period of time.
 5. The data processing deviceaccording to claim 4, wherein a detection of dozing of the target personis performed when the ratio is out of a range between a predeterminedlower limit threshold value excluding 0 and a predetermined upper limitthreshold value excluding infinity.
 6. The data processing deviceaccording to claim 1, wherein the set of instructions further causes thedata processing device to: create an eye opening degree transition graphon the basis of the determined propriety of the eye opening degree;calculate an average eye opening period of time on the basis of the eyeopening degree transition graph; and determine shutter time on the basisof the average eye opening period of time and the eye opening degree toallow an imaging device to image.
 7. An eye opening degree detectionmethod of a data processing device that detects an eye opening degreebased on first and second images that include, at least, regions of botheyes of a target person and that are obtained by being imaged whileshifting time, the method comprising: calculating each one eye openingdegree of the left and right eyes of the first image; creating a pixelvalue histogram of each region of the left and right eyes when adifference between the left and right one eye opening degrees does notfall within a predetermined ratio, and selecting one of the left andright one eye opening degrees based on the pixel value histogram;calculating an eye opening degree based on the left and right one eyeopening degrees when the difference between the left and right one eyeopening degrees falls within the predetermined ratio, and calculatingthe eye opening degree based on the selected one eye opening degree whenthe difference between the left and right one eye opening degrees doesnot fall within the predetermined ratio; and determining a propriety ofthe eye opening degree calculated based on the first image by comparingthe eye opening degree calculated based on the first image with an eyeopening degree calculated based on the second image prior to the firstimage.
 8. The eye opening degree detection method according to claim 7,further comprising comparing the calculated left and right one eyeopening degrees of the first image to identify a larger one eye openingdegree, wherein the predetermined ratio comprises a predetermined ratioto the identified larger one eye opening degree.
 9. The eye openingdegree detection method according to claim 7, further comprisingdetermining that the eye opening degree calculated on the basis of thefirst image is valid when a difference between the eye opening degreecalculated on the basis of the first image and the eye opening degreecalculated on the basis of the second image is equal to or smaller thana threshold value calculated on the basis of an imaging timingdifference between the first image and the second image.
 10. The eyeopening degree detection method according to claim 7, furthercomprising: creating an eye opening degree transition graph on the basisof the determined propriety of the eye opening degree; determining aneye opening period of time and an eye closing period of time on thebasis of the eye opening degree transition graph; and detecting dozingof the target person on the basis of a ratio of the eye opening periodof time to the eye closing period of time in a predetermined period oftime.
 11. The eye opening degree detection method according to claim 10,wherein a detection of dozing of the target person is performed when theratio is out of a range between a predetermined lower limit thresholdvalue excluding 0 and a predetermined upper limit threshold valueexcluding infinity.
 12. The eye opening degree detection methodaccording to claim 7, further comprising: creating an eye opening degreetransition graph on the basis of the determined propriety of the eyeopening degree; calculating an average eye opening period of time on thebasis of the eye opening degree transition graph; and determiningshutter time on the basis of the average eye opening period of time andthe eye opening degree to allow an imaging device to image.